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During  the  period  of  this  report  a  document  entitled:  "Biomorphic  Networks:  Approach  to 
invariant  feature  extraction  and  segmentation  for  ATR"  was  completed.  It  consolidates  and 
describes  earlier  and  more  recent  efforts  in  this  aspect  of  our  research.  A  copy  of  the 
document  is  attached  to  serve  as  the  quarterly  report.  A  paper  with  the  same  title  is 
scheduled  for  presentation  at  SPIE'98,  San  Diego  this  July. 
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1.  N.H.  Farhat,  "Biomorphic  Dynamical  Networks  for  Cognition  and  Control", 
Journal  of  Intelligent  and  Rototic  Systems,  Vol.  21,  pp.  167-177,  (1998).  (copy 
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Biomorphic  networks:  Approach  to  invariant 
feature  extraction  and  segmentation  for  ATR* 


A.  S.  Baek  and  N.  H.  Farhat 
Department  of  Electrical  Engineering 
University  of  Pennsylvania 
Philadelphia,  PA  19104 


ABSTRACT 

Invariant  features  in  two  dimensional  binary  images  are  extracted  in  a  single  layer  network  of  locally  coupled  spiking 
(pulsating)  model  neurons  with  prescribed  synapto-dendritic  response.  The  feature  vector  for  an  image  is  represented 
as  invariant  structure  in  the  aggregate  histogram  of  interspike  intervals  obtained  by  computing  time  intervals  between 
successive  spikes  produced  from  each  neuron  over  a  given  period  of  time  and  combining  such  intervals  from  all  neurons 
in  the  network  into  a  histogram.  Simulation  results  show  that  the  feature  vectors  are  more  pattern-specific  and 
invariant  under  translation,  rotation,  and  change  in  scale  or  intensity  than  achieved  in  earlier  work.  We  also  describe 
an  application  of  such  networks  to  segmentation  of  line  (edge-enhanced  or  sihlouette)  images.  The  biomorphic  spiking 
network’s  capabilities  in  segmentation  and  invariant  feature  extraction  may  prove  to  be,  when  they  are  combined, 
valuable  in  Automated  Target  Recognition  (ATR)  and  other  automated  object  recognition  systems. 

Keywords:  Spiking  neural  networks,  biomorphic  neuron,  feature  extraction,  distortion  invariance,  segmentation, 
automated  object  recognition 


1.  INTRODUCTION 

Invariant  feature  extraction  plays  a  central  role  in  designing  automatic  object  recognition  systems,  whose  applications 
permeate  a  diverse  range  of  fields  that  include  radar,  sonar,  optical  character  recognition,  autonomous  vision  machines 
for  use  in  space  exploration,  robotics,  and  manufacturing.  The  human  visual  system  recognizes  objects  remarkably 
well  from  two  dimensional  images  cast  on  the  retina,  even  when  these  images  are  distorted  by  various  factors  such 
as  change  in  the  distance  and  orientation  of  the  object,  light  illumination  level,  shading  and  background  clutter. 
Understanding  and  duplicating  this  exceptional  invariant  recognition  ability  of  the  visual  system  is  valuable  for 
successful  development  of  advanced  automatic  target  recognition  (ATR)  systems. 

To  this  end  artificial  neural  network  models  based  on  neuronal  firing  rate  have  been  explored  and  applied  in  the 
past,  with  various  degrees  of  success,  to  both  the  feature  extraction  and  the  recognition  aspects  of  ATR  [1].  It  is 
however  becoming  increasingly  clear  that  the  temporal  aspects  of  brain  function  manifested  in  the  spiking  nature  of 
neural  activity  and  the  relative  timing  of  spikes  may  play  an  important  role  in  distortion  invariant  feature-extraction, 
feature-binding  and  other  higher-level  brain  functions. 

In  this  paper  we  will  present  methods  for  invariant  feature  extraction  and  image  segmentationin  using  a  network 
of  biomorphic  spiking  neurons  that  extends  and  improves  on  earlier  published  work.  Invariant  features  in  two 
dimensional  binary  images  are  extracted  in  a  single  layer  network  of  locally  coupled  spiking  (pulsating)  neurons 
that  include  synapto-dendritic  processing.  The  feature  vector  for  an  image  emerges  as  invariant  structure  in  the 
aggregate  histogram  of  interspike  intervals,  which  is  formed  by  computing  time  intervals  between  successive  spikes 
produced  from  each  neuron  over  a  given  period  of  time  and  combining  such  intervals  from  all  neurons  in  the  network 
into  a  histogram.  The  simulation  results  presented  here  show  that  the  feature  vectors  are  object-specific  and  highly 
invariant  under  translation,  rotation,  and  change  in  scale  or  intensity.  The  results  suggest  that  the  combined  process 


*  Based  in  part  on  an  oral  presentation  given  given  at  the  1997  OS  A  Annual  Meeting,  Long  Beach,  CA  Oct.  1997. 


of  the  segmentation  and  feature  extraction  from  segmented  images  may  provide  useful  basis  in  designing  ATR  and 
other  autonomous  recognition  systems. 

Current  techniques  for  distortion  invariant  pattern  recognition  use  integral  transforms,  algebraic  moments,  or 
perceptron-like  neural  networks  with  learning  algorithms  [2],  These  techniques  demand,  however,  extensive  com¬ 
putational  resource  and  are  susceptible  to  spatial  noise  in  the  input  image.  Recently,  a  novel  pulse-coupled  neural 
network,  which  could  generate  invariant  signatures  of  images  of  simple  canonical  patterns  under  translation,  rotation, 
scale  and  intensity  changes,  was  described  in  [3].  Although  our  method  is  an  extension  of  this  earlier  work  in  that 
it  also  uses  a  pulse-coupled  neural  network  and  employs  concepts  developed  in  [3]  and  [4],  there  is  a  fundamental 
difference  between  the  network  dynamics  in  the  two  methods.  The  method  presented  in  [3]  is  based  on  the  cortical 
model  of  synchronization  of  neural  temporal  activity  developed  in  [4]  but  uses  moderate-to-weak  linking  strengths 
to  induce  phase-locked  firing  states  of  neurons.  Also  in  [3],  the  spatial  structure  of  the  test  patterns  (a  cross  and 
a  tee),  made  of  blocks  with  each  block  containing  11  x  11  pixels  and  a  distinctive  gray-scale  intensity  level,  was 
encoded  in  the  phase-locking  pattern  of  output  spikes  from  neurons.  In  contrast,  the  method  described  here  is  less 
“corticomorphic” .  It  is  based  on  firing  rate  encoding,  which  is  predominantly  observed  in  sensory  nervous  systems, 
rather  than  phase-locking,  and  is  suited  for  processing  binary  line  images.  This  different  encoding  scheme  arises,  as 
explained  below,  from  the  opposite  relationship  between  the  time  constants  of  the  synaptic  response  function  and  the 
membrane  potential  (pulse  generator)  that  we  use  as  compared  to  [3].  As  a  result,  the  method  appears  to  produce 
invariant  feature  vectors  that  are  more  input  pattern  specific  than  in  [3],  which  is  a  desirable  prerequisite  for  accurate 
classification.  Also,  its  ability  to  extract  invariant  features  of  binary  line  images  is  an  attractive  attribute  because 
edge-enhancement  and  line  extraction  are  standard  operations  in  pattern  recognition. 

In  Section  4,  we  demonstrate  the  ablity  of  the  spiking  neural  networks  in  segmentation  of  line  images  (eg. 
silhouettes  or  edge  enhanced  images)  of  model  objects.  Once  characteristic  segments  of  a' line  image  are  obtained, 
each  segment  may  be  processed  by  a  biomorphic  spiking  network,  similar  to  the  one  described  in  Section  2,  to  produce 
a  set  of  histograms  of  interspike  intervals,  which  contain  invariant  features  that  are  specific  to  the  line  image.  The 
resulting  set  of  invariant  features,  which  belong  to  the  respective  features  of  the  image,  can  then  be  regarded  as  a 
composite  invariant  feature  vector  that  represents  the  entire  line  image. 

2.  THE  NETWORK 

In  the  invariant  feature  extraction  system  presented  here,  a  basic  image  that  consists  of  a  12  x  12  binary  pixel  array 
is  fed  to  a  single  layer  network  that  consists  of  spiking  neurons  arranged  in  the  12  x  12  array  format.  The  model 
neuron  used  in  the  network  is  of  the  integrate- an d-fire  (I&F)  variety  ,  similar  to  that  used  in  [3]- [5],  and  is  regarded  as 
biomorphic  in  that  it  contains  a  simplified  model  of  the  dendritic-tree  processing  in  biological  neurons,  as  explained 
below. 

The  dynamics  of  the  neuron  can  be  summarized  as  follows,  refering  to  Fig.l.  When  the  exponentially  rising 
membrane  potential,  A Vm(t),  of  the  neuron  at  (i,  j)  reaches  the  time- varying  threshold,  V-fit),  it  instantaneously 
drops  to  the  resting  membrane  potential  Vrest  and  at  the  same  time  a  spike  (action  potential)  is  generated  [6].  The 
time-varying  threshold  is  represented  as 

v${t)  =  v0-uij{t)  (1) 

where  V0  is  constant  and  Uij(t),  as  defined  below  in  Eqn.  (3),  is  a  signal  that  results  from  the  joint  actions  of 
synaptic  inputs  and  the  intensity  of  the  image.  This  process  of  exponential  rise  and  instantaneous  drop  of  the 
membrane  potential  repeats,  generating  successive  spikes  whose  timing  is  modulated  by  the  threshold  voltage.  Note 
that  lowering  the  threshold  causes  the  neuron  to  fire  faster,  because  it  takes  less  time  to  build  the  membrane  potential 
from  the  resting  potential  to  the  threshold. 

In  an  effort  to  emulate  the  signal  processing  that  occurs  in  the  dendritic  tree  of  biological  neurons,  the  model 
neuron  is  given  a  simplified  homogeneous  synapto-dendritic  response  function  modeled  as  an  impulse  response  func¬ 
tion,  which  approximates  the  change  in  membrane  potential  (depolarization)  at  the  neuron’s  hillock  (i.e,  the  site 
of  output  spike  generation),  in  response  to  a  single  spike  on  its  dendritic  tree.  We  use  an  exponentially  decaying 
function  for  the  synapto-dendritic  response,  given  by 


h(t)  =  /i0exp(— t/r) 


(2) 


S  ij(t)  =  I  ij(l+  pPij(t» 
P  y  (t)  =  ST  ij  *  h  (t) 
h  (t)  =  h  Qexp(  -t  /  x) 


(X  =  1  msec  (membrane  time  constant) 
x  =  12  msec  (synapto-dendritic  time  constant) 


Figure  1.  The  dynamics  of  the  biomorphic  spiking  neuron  and  parameters  used  in  numerical  simulations. 

where  r  is  the  time  constant  of  the  response.  Also,  the  model  neuron  in  the  network  receives  the  intensity  value  of 
the  image  at  the  corresponding  coordinates,  which  modulates  the  signal  produced  from  synaptic  inputs  in  a  nonlinear 
manner  similar  to  that  in  [3]  and  [4],  as  further  explained  below. 

Using  the  same  notation  as  in  [3]  and  [4],  the  signal,  (%(£),  which  is  produced  from  a  nonlinear  combination 
of  the  external  input  and  synaptic  inputs,  is  determined  by  Eqn.  (3)  with  i  and  j  representing  the  coordinates  of 
both  the  pixel  and  the  neuron  in  the  12  x  12  array.  The  signal  Pij(t),  as  defined  in  Eqn.  (4),  is  a  convolution  of 
the  synapto-dendritic  response  function  and  the  input  spike  train  impinging  on  the  neuron  at  The  input  spike 

train  is  represented  below  as  a  sum  of  delta  functions  with  m  and  n  representing  the  coordinates  of  the  sending 
neuron  and  k  representing  the  spike  generation  time. 

=  Ii3  x  (1  +  pPijit)) 

Pij(t)  =  STij  *  h(t) 

STij  (t)  = 

mnk 

The  signal  Sy(f )  modulates  the  threshold  voltage  of  the  neuron  at  (i,j),  as  shown  in  Eqn.  (1),  and  this  will  determine 
the  firing  times.  The  departure  of  the  membrane  potential  Vm  from  the  resting  value  Vrest  can  be  expressed  as: 

A  Vm{t)  =  E{  l-exp(— ^))  (6) 

E  =  V0-e  (7) 

where  V0  and  e  are  given  values  of  5  and  0.2  volts,  respectively  in  the  simulations  below.  When  a  spike  is  elicited,  the 
membrane  potential  Vm  instantaneously  drops  to  the  resting  potential  Vrest  and  begins  to  rise  exponentially  towards 
the  time- varying  threshold,  V^it)  as  depicted  in  Fig.l. 

Attention  is  drawn  to  the  relationship  between  the  time  constant  /i  of  the  exponential  membrane  build-up  and 
the  time  constant  r  of  the  synapto-dendritic  response  function.  In  contrast  to  the  practice  in  [3]  in  which  the  time 

constant  of  the  synapto-dendritic  response  is  shorter  than  that  of  the  membrane  dynamics  (referred  to  as  pulse 

generator  in  [3]),  the  time  constant  r  of  the  synapto-dendritic  response  here  is  longer  than  the  time  constant  fi  of 
the  membrane  potential  build-up.  This  alteration  of  the  relation  between  the  time  constants  produces  a  markedly 
different  network  behavior  in  that  the  prolonged  synapto-dendritic  response  causes  slow  depressions  in  the  threshold 
of  a  neuron,  affecting  its  firing  rate.  In  our  simulations,  the  time  constant  r  of  the  synapto-dendritic  response  function 
is  12msec,  while  that  of  the  membrane  potential  build-up  is  1msec.  In  contrast  the  time  constants  in  [3]  are  1  for 
synaptic  response  and  5  for  the  membrane  threshold  (pulse  generator)  (units  were  not  specified). 

Another  important  feature  of  the  model  neuron  used  here,  which  is  adopted  from  [3]  and  [4],  is  the  nonlinear 
combination  of  the  extrinsic  input  Uj  and  the  intrinsic  input  from  other  neurons  as  defined  by  Eqn.  (3).  The 


(4) 

(5) 


pixel  intensity  of  the  image  modulates  the  signal  produced  from  synaptic  inputs.  The  strength  of  the  modulation 
is  controlled  by  the  constant  /3.  Especially,  when  Jy  is  zero,  the  neuron  at  (i,  j)  can  not  fire,  because  Uij(t )  then 
becomes  zero,  according  to  Eqn.  (3) ,  and  the  neuron  operates  in  the  sub-threshold  mode  (the  membrane  potential 
saturates  before  reaching  the  threshold  voltage).  The  value  of  Jy  =  0  effectively  nullifies  any  effects  of  synaptic 
inputs  on  the  threshold,  forcing  the  neuron  to  be  “silent”.  This  nonlinear  operation  is  similar  to  the  biological 
situation  in  which  inhibitory  inputs  proximal  to  the  soma  (cell  body)  may  effectively  nullify  excitation  by  inputs  on 
distal  synapses  [7].  In  a  complex  morphological  dendritic  tree,  this  type  of  a  selective  AND-NOT  like  operation  can 
effectively  decouple  subunits  of  the  tree. 

The  network  architecture  used  is  shown  in  Fig.  2(a).  The  neurons  are  arranged  in  a  12  x  12  array  and  a 
sample  neuron  in  the  lower  right  side  of  the  array  is  depicted  showing  its  synaptic  connections  from  its  8  nearest 
neighboring  neurons.  Interspike  intervals  of  the  spike  train  from  each  neuron  are  computed  in  the  local  interspike 
interval  analyzers  and  the  results  are  combined  to  form  the  aggregate  interspike  interval  histogram.  Shown  in  Fig. 
2(b)  is  the  schematic  of  an  isolated  neuron  with  the  two  types  of  inputs,  the  instrinsic  synaptic  inputs  from  the 
neighboring  neurons  and  the  extrinsic  pixel  intensity  value  from  the  image. 

3.  SIMULATION  RESULTS 

The  simulations  results  are  shown  in  Figs.  3,  4,  and  5  with  the  input  images  and  the  associated  aggregate  (interspike 
interval  histograms)  ISIHs  shown  in  the  left  and  the  right  columns,  respectively.  Figure  3(a)  shows  the  simulation 
result  with  the  original  image  of  the  tee.  The  input  image  in  Fig.  3(b)  is  a  scaled,  90-degree  rotated,  and  translated 
version  of  the  original  tee  image.  Figures  3(c)  and  (d)  show  the  input  images  of  the  cross  and  the  associated  output 
ISIHs.  Already  from  these  simulations,  one  can  clearly  see  the  differences  between  the  histograms  produced  from  the 
tee  images  and  from  the  cross  images  and  the  invariance  of  the  histograms  with  image  distortions.  In  Figs.  4(a)  and  (b) 
for  a  rectangle,  the  histograms  are  drastically  different  from  the  previous  cases  of  the  tee  and  the  cross,  highlighting 
the  pattern-specificity,  and  the  invariance  is  well  retained  for  the  rectangle  in  the  distortions  of  scale,  rotation,  and 
translation.  As  reader  might  have  noticed,  the  angles  used  in  the  simulations  for  the  rotational  invariance  are  confined 
to  the  multiples  of  90  degrees,  due  to  the  limited  spatial  resolution.  At  higher  spatial  resolution,  this  restriction  on 
the  rotation  angle  can  be  relaxed  and  the  rotation  invariance  for  in-between  angles  will  be  achieved.  The  invariance 
with  scale  for  a  triangle  is  shown  in  Fig.  4(c)  and  (d).  We  have  also  examined  invariance  with  intensity  for  the 
triangle,  as  demonstrated  in  Figs.  5  (a)  and  (b).  The  uniform  intensity  of  the  triangle  used  for  Figs.  5  (a)  is  10 
percent  higher  than  that  used  for  Figs.  5  (b).  The  characteristic  form  of  the  histogram  for  the  triangle  is  seen  to  be 
retained  but  experiences  a  shift  towards  lower  time  interval  region.  The  shift  comes  from  the  fact  that  the  higher 
uniform  image  intensity  lowers  the  thresholds  of  the  neurons,  raising  the  firing  rate.  On  a  relevant  note,  we  have 
an  indication  from  later  simulations  that  using  a  synapto-dendritic  response  function  that  is  more  biomorphic,  i.e., 
approximates  the  biological  response  more  accurately  [8],  results  in  higher  invariance  to  the  distortions,  especially 
the  intensity  change.  Results  from  these  simulations  are  not  reported  here  due  to  space  limitation. 

The  invariant  structure  in  ISIH  is  closely  related  to  characteristic  local  features  of  the  input  pattern.  For  instance, 
the  main  feature  of  the  triangle  is  the  three  corners.  The  neurons  residing  in  these  corners  of  the  triangle  and  the 
vicinities  have  higher  firing  rates  that  are  responsible  for  invariant  structure  in  the  ISIH,  since  they  are  connected  to 
more  neighboring  neurons  with  higher  activity  than  the  neurons  in  the  middle  of  edges.  Such  analysis  carried  out  by 
the  network  is  meaningful  for  syntactical  information  processing  in  that  the  input  image  can  be  implicitly  segmented 
into  sections  that  contain  characteristic  structural  information.  It  is,  however,  important  to  remember  that  due  to 
the  convoluted  nature  of  inter-neuron  interactions  through  pulse-coupling,  the  invariant  structure  in  the  ISIH  is  also 
affected  indirectly  by  activity  of  neurons  that  are  beyond  the  nearest  neighboring  connections.  Although  this  distal 
interaction  is  much  weaker  than  the  local  interaction,  it  still  seems  to  take  a  part  in  determining  the  shape  of  the 
ISIH. 


4.  SPIKING  NETWORK  FOR  SEGMENTATION 

A  preliminary  study  of  the  ability  of  biomorphic  spiking  networks  to  segment  line  images  (i.e.,  silhouettes  or  edge 
enhanced  images)  of  model  objects  was  carried  out.  The  representative  results  given  next  suggest  that  combining 
spiking  networks  for  segmentation  with  spiking  networks  for  invariant  feature  extraction  may  offer  a  viable  approach 
for  generating  invariant  features  for  extended  objects. 


(a) 


(b) 


spike 
output  to 
neighboring 
neurons 


local  interspike  interval  analyzer 


inputs 

from  / 

neighboring 
neurons  analog  input  from  the  image 


Figure  2.  (a)  Schematic  of  the  network  arranged  in  the  N  x  N  array.  A  set  of  exemplar  synaptic  connections 
of  a  neuron  from  its  eight  neighboring  neurons  is  shown  in  the  lower  right  side  of  the  array,  with  the  cursive  lines 
representing  the  connections.  Interspike  intervals  from  each  neuron  are  computed  in  the  local  interspike  interval 
analyzer  and  sent  to  the  aggregate  interspike  interval  histogram,  (b)  Schematic  of  the  neuron  with  two  types  of 
inputs:  synaptic  inputs  from  its  neighboring  neurons  and  an  analog  input  of  the  image  intensity  at  the  neuron’s 
location.  The  two  types  of  the  inputs  are  combined  in  a  multiplicative  manner  to  produce  a  signal  that  influences 
the  timing  of  action  potential  generation  and  therefore  the  spiking  output  to  the  synaptically  connected  neurons. 


Figure  3.  Simulation  results  with  the  tee  and  the  cross  demonstrating  invariance  with  translation,  rotation,  and 
scale.  The  input  images  and  the  aggregate  (interspike  interval  histograms)  ISIHs  from  simulations  are  shown  in  the 
left  and  the  right  column,  respectively,  (a)  The  tee  image,  (b)  A  90-degree  rotated,  scaled,  and  translated  version  o 
the  tee  in  (a),  (c)  The  cross  image,  (d)  A  scaled  and  translated  version  of  the  cross  m  (c).  The  time  resolution  ot 

the  ISIHs  is  0.001  msec. 


In  order  to  obtain  the  input  line  images,  test  objects  (two  types  of  model  aircrafts  and  one  tank)  were  photographed 
using  a  CCD  camera,  then  digitized  and  saved  in  gray-scale  in  400  x  400  pixel  format.  Examples  of  such  gray  sca  e 
images  are  shown  in  Figs.  6(a),  7(a),  and  8(a).  The  gray-scale  images  were  then  binarized  and  edge-detected  to 
produce  the  line  images  as  shown  in  Figs.  6(b),  7(b),  and  8(b).  In  producing  the  line  image,  if  the  gray-scale  image 
is  smaller  than  400  x  400,  we  padded  the  remaining  pixels  with  zero  intensity  to  maintain  a  consistent  image  size. 
Then  the  line  image  was  fed  into  the  biomoprhic  neural  network,  which  has  the  same  dimensions  as  those  ot  the 
input’ image  format.  The  neurons  of  the  network  used  here  employed  an  alpha  function  [8]  for  the  synapto-dendntic 
response  in  place  of  the  exponential  function  given  by  Eqn.3.  Note  that  the  neurons  which  receive  zero  intensity 
from  the  image  do  not  fire,  because  the  signal  Sij(t)  in  Eqn.  (3)  is  zero  for  all  such  neurons. 

Figures  6(c),  7(c),  and  8(c)  present  the  simulation  results  with  the  neurons’  firing  rates  given  in  gray  scale,  the 
synaptic  weight  matrix  used  in  the  simulation  was  circularly  symmetric  with  a  radius  of  21  pixels,  and  had  a  uniform 
excitatory  synaptic  weight  of  one  within  the  circle.  In  these  plots,  the  brighter  pixels  indicate  higher  firing  rates  or 
the  neurons  located  in  those  spots.  We  then  varied  a  threshold  firing  rate  and  plotted  only  those  pixels  whose  firing 
rate  is  above  the  threshold,  as  shown  in  Figs.  6(d),  7(d),  and  8(d)  to  segment  the  line  images  into  characteristic 
segments  each  of  which  can  potentially  be  represented  by  an  invariant  ISIH  with  the  aid  of  a  network  similar  to  t 
described  in  Section  2.  The  pronounced  features  in  Fig.  5  6(d)  are  the  engines  in  the  rear,  the  tips  of  the  wings 
the  angels  at  the  functions  of  the  wings  and  the  main  body,  while  in  Fig.  7(d)  they  are  the  tail,  the  engines  and  the 
gun  barrels  on  the  wings,  and  the  front  tip.  In  Fig.  6(d),  the  biomoprhic  network  extracted  (segmented)  the  most 
characteristic  feature  of  tanks,  the  nozzle. 

Figures  9  and  10  show  the  invariance  of  segmenting  the  line  images  of  Aircrafts  1  and  2  under  translation 
rotation,  and  change  of  scale.  The  simulation  results  using  images  of  Aircrafts  1  and  2  are  shown  with  the  inpu 
edge-enhanced  images  given  on  the  left  and  the  simulation  results  showing  neurons  with  firing  rates  above  a  certain 
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Figure  4.  Simulation  results  with  the  rectangle  and  the  triangle  demonstrating  invariance  with  invariance  of 
translation,  rotation,  and  scaling.  The  input  images  and  the  aggregate  ISIHs  from  simulations  are  shown  in  the  left 
and  the  right  columns,  respectively,  (a)  The  rectangle  image,  (b)  A  90-degree  rotated,  scaled,  and  translated  version 
of  the  rectangle  in  (a),  (c)  The  triangle  image,  (d)  A  scaled  version  of  the  triangle  in  (c). 
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Figure  5.  Simulation  results  with  the  triangle  for  intensity  invariance.  The  input  images  and  the  aggregate  ISIHs 
from  simulations  are  shown  in  the  left  and  the  right  columns,  respectively,  (a)  The  triangle  image,  (b)  The  triangle 
image  with  uniform  intensity  increased  by  10  percent. 
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Figure  6.  Segmentation  of  gray  scale  image  of  model  Aircraft  1.  (a)  Gray  scale  image  of  model  Aircraft  1,  (b) 
The  line  image  of  the  gray  scale  image  in  (a),  (c)  Firing  rates  of  the  biomorphic  neurons  with  the  brightest  pixels 
representing  the  highest  firing  rates,  (d)  Plot  of  firing  rates  above  a  prescribed  threshold.  The  plot  shows  characteristic 
segments  of  the  aircraft,  each  of  which  maybe  further  analyzed  by  a  feature  extracting  network  to  represent  it  by 
an  invariant  ISIH.  The  invariant  histograms  of  the  segments  form  an  “invariant  set” ,  which  can  be  used  to  represent 
the  object  and  serve  as  an  input  to  an  ATR  system. 

threshold  in  the  right.  In  these  simulations,  the  circularly  symmetric  synaptic  weight  matrix  has  two  concentric 
circles  with  radii  of  48  and  51  pixels.  The  matrix  elements  inside  the  inner  circle  with  radius  of  48  pixels  have 
a  uniform  excitatory  synaptic  weight  of  one,  while  those  between  the  inner  and  the  outer  circles  have  a  uniform 
inhibitory  synaptic  weight  of  negative  one.  We  observed  that  the  circularly  symmetric  weight  with  a  large  radius 
produces  better  rotation  invariance.  When  the  radius  decreases,  the  synaptic  matrix  delineated  over  the  rectangular 
array  loses  circular  symmetry  and  becomes  a  square,  due  to  the  spatial  resolution  limitation.  Also  in  the  simulations 
shown  in  Figs.  9  and  10,  the  inhibitory  weight  is  used  to  balance  off  the  excitatory  synaptic  connections  and  avoid 
saturation  of  neurons’  firings.  The  simulation  results  show  that  robust  invariance  under  translation,  rotation,  and 
scale,  is  achieved. 

In  the  above  simulations  involving  model  objects,  as  well  as  in  the  previous  examples  involving  canonical  objects, 
the  geometrical  complexities  are  encoded  in  firing  rates  of  the  neurons  that  reside  in  the  corresponding  locations.  As 
we  have  seen,  sections  of  the  images  that  contain  complicated  structure  become  pronounced  in  the  firing  rate  plot. 

5.  CONCLUSIONS 

We  have  described  a  spiking  neural  network  that  can  extract  invariant  feature  vectors  of  two  dimensional  binary  line 
images  based  on  firing  rate  encoding  rather  than  encoding  by  phase-locking  described  in  [3].  The  difference  seems 
to  lead  to  feature  vectors  or  signatures  for  binary  line  images  that  are  more  pattern-specific  and  invariant  under 
translation,  rotation,  and  change  in  scale  or  intensity  than  achieved  in  [3].  Also  the  simpler  neuronal  and  network 
structures  used  here  can  be  advantageous  in  hardware  implementation  for  real-time  processing. 

A  preliminary  study  of  the  ability  of  biomophic  spiking  networks  to  segment  line  images  (i.e.,  silhouettes  or  edge 
enhanced  images)  of  model  objects  was  also  carried  out.  The  results  suggest  that  the  the  combined  processing  of 
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Figure  7.  Segmentation  of  gray  scale  image  of  model  Aircraft  2.  (a)  Gray  scale  image  of  model  Aircraft  2,  (b) 
The  line  image  of  the  gray  scale  image  in  (a),  (c)  Firing  rates  of  the  biomorphic  neurons  with  the  brightest  pixels 
representing  the  highest  firing  rates,  (d)  Plot  of  firing  rates  above  threshold.  The  plot  shows  characteristic  segments 
of  the  aircraft. 

segmentation  and  feature  extraction,  each  peformed  by  respective  spiking  networks,  may  provide  a  viable  approach 
for  generating  invariant  feature  vectors  for  extended  objects,  which  is  worthy  of  further  investigation. 
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Figure  8.  Segmentation  of  gray  scale  image  of  a  model  tank,  (a)  Gray  scale  image  of  a  model  tank,  (b)  The  line 
image  of  the  gray  scale  image  in  (a) ,  (c)  Firing  rates  of  the  biomorphic  neurons  with  the  brightest  pixels  representing 
the  highest  firing  rates,  (d)  Plot  of  firing  rates  above  a  threshold.  The  plot  shows  a  characteristic  segment  of  the 
tank,  the  nozzle. 
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Figure  9.  Invariant  segmentation  of  edge-enhanced  images  of  model  Aircraft  1.  The  input  images  and  the  invariant 
segmented  image  formed  by  thresholding  firing  rates  of  the  neurons  are  shown  in  the  left  and  the  right  columns, 
respectively,  (a)  Segmentation  of  the  original  edge  enhanced  image  of  model  Aircraft  1,  (b)  Segmentation  of  a  45- 
degree  rotated  and  translated  image  of  the  original  image  shown  in  (a),  (c)  Segmentation  of  a  90-degree  rotated  image 
of  the  original,  (d)  segmentation  of  a  90-degree  rotated,  80-percent  scaled,  and  translated  image  of  the  original. 
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Figure  10.  Invariant  segmentation  of  edge-enhanced  images  of  model  Aircraft  2.  The  input  images  and  the  invariant 
features  derived  from  thresholding  firing  rates  of  the  neurons  are  shown  in  the  left  and  the  right  columns,  respectively, 
(a)  Segmentation  of  the  original  edge  enhanced  image  of  model  Aircraft  2,  (b)  Segmentation  of  a  135-degree  rotated 
and  translated  image  of  the  original  image  shown  in  (a),  (c)  Segmentation  of  a  translated  image  of  the  original, 
(d)  Segmentation  of  a  90-degree  rotated  and  120-percent  scaled  image  of  the  original.  Notice  a  change  of  scale  of 
a  segment  will  not  affect  the  ability  of  a  spiking  network  of  the  type  described  in  Section  2  to  extract  an  invariant 
IHIS. 
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Abstract.  Advances  in  understanding  the  neuronal  code  employed  by  cortical  networks  indicate 
that  networks  of  parametrically  coupled  nonlinear  iterative  maps,  each  acting  as  a  bifurcation 
processing  element,  furnish  a  potentially  powerful  tool  for  the  modeling,  simulation,  and  study  of 
cortical  networks  and  the  host  of  higher-level  processing  and  control  functions  they  perform.  Such 
functions  are  central  to  understanding  and  elucidating  general  principles  on  which  the  design  of 
biomorphic  learning  and  intelligent  systems  can  be  based.  The  networks  concerned  are  dynamical 
in  nature,  in  the  sense  that  they  “compute”  not  only  with  static  (fixed-point)  attractors  but  also 
with  dynamic  (periodic  and  chaotic)  attractors.  As  such,  they  compute  with  diverse  attractors,  and 
utilize  transitions  (bifurcation)  between  attractors  and  transient  chaos  to  carry  out  the  functions  they 
perform.  An  example  of  a  dynamical  network,  a  parametrically  coupled  net  of  logistic  processing 
elements,  is  described  and  discussed  together  some  of  its  behavioural  attributes  that  are  relevant  to 
elucidating  the  possible  role  for  coherence,  bifurcation,  and  chaos  in  higher-level  brain  functions 
carried  out  by  cortical  networks. 

Key  words:  neuronal  code,  netlets,  logistic  map,  dynamical  computing,  coherence,  bifurcation, 
chaos,  collapse  into  low-dimensional  attractors. 


1.  Introduction 

Meaningful  progress  in  learning  and  intelligent  systems  is  difficult  without  ability 
to  model  and  simulate  cortical  dynamics  and  the  way  cortical  networks  carry  out 
the  amazing  cognitive  and  control  functions  we  humans  seem  to  do  so  effort¬ 
lessly  well  while  coping  and  learning  in  a  complex  environment.  The  cortex, 
also  believed  to  be  the  site  of  conciousness  and  all  higher-level  thought  pro¬ 
cesses,  is  fundamentally  a  high-dimensional  nonlinear  dynamical  system.  It  is 
well  known  that  a  dynamical  system  of  dimension  greater  than  two  (TV  ^  3) 
can  exhibit  in  its  state-space,  depending  on  the  range  of  its  control  parameters, 
all  three  types  of  primary  attractors:  fixed-point,  periodic,  and  chaotic.  Attrac¬ 
tor  type  neural  networks  in  use  today  to  model  brain  function,  rely  entirely  on 
point-attractors  in  their  operation.  Therefore  they  can  not  perform  any  higher- 
level  cortical  functions  that  might  depend  on  periodic  or  chaotic  attractors  or  on 
bifurcations  between  attractors.  To  gain  insight  in  cortical  dynamics  and  in  the 
general  principles  underlying  learning  and  intelligence,  one  needs  to  develop  a 
new  class  of  neural  net  models  capable  of  “computing”  with  diverse  attractors, 
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and  elucidate  the  roles  of  coherence  (periodicity,  synchronicity,  phase-locking), 
bifurcation,  complexity,,  and  chaos  in  the  kind  of  analog  “computations”,  and 
autonomous  learning  they  may  carry  out.  To  do  this  meaningfully,  one  needs 
to  consider  one  of  the  most  fundamental  questions  in  neuroscience  and  brain 
research  today,  namely  the  nature  or  identity  of  the  neuronal  code,  i.e.,  the  way 
the  cortex  deciphers  and  encodes  information  as  it  carries  out  cognitive,  learn¬ 
ing,  and  control  functions.  An  attempt  at  elucidating  the  neuronal  code  and  how 
it  suggests  that  cortical  networks  may  be  modeled,  simulated,  and  studied  with 
the  help  of  parametrically  coupled  nets  of  logistic  processing  elements  is  given 
in  Section  2.  This  is  followed  in  Section  3  by  description  of  some  of  the  more 
interesting  behavior  we  have  observed  in  simulations  carried  out  with  such  a  net¬ 
work  under  uniform  global  coupling  and  external  stimulus  with  spatio-temporal 
input.  It  is  found  that  such  networks  have  the  capacity  to  “self-anneal”  or  col¬ 
lapse  into  stimulus  (input)  dependent  periodic  (period-m)  attractors  following  a 
transient  “chaotic”  period  during  which  the  network  searches  its  state-space  for 
an  associated  dynamic  attractor  of  low  dimension.  The  network  accepts  naturally 
both  time-varying  or  stationary  input  patterns.  Moreover  we  find  that  the  use  of 
activity  dependent  nonlinear  quantized  coupling  strengths,  provides  the  network 
with  clustering  ability  wherein,  depending  on  the  stimulus  pattern,  PEs  in  the 
network  divide  into  phase-locked  groups  with  the  PEs  within  each  group  pos¬ 
sessing  synchronized  identical  period-m  orbits.  The  value  of  m  is  found  to  be 
the  same  for  all  clusters  and  the  number  of  clusters  gives  the  dimension  of  the 
periodic  attractor.  The  implications  of  these  findings  for  higher-level  processing 
such  as  feature-binding  and  development  of  novel  learning  algorithms  are  briefly 
discussed. 


2.  The  Neuronal  Code 

There  is  much  evidence,  stemming  from  anatomical,  physiological,  and  modeling 
work  in  favour  of  the  hypothesis  that  the  basic  functional  unit  in  the  cortex  is  the 
neuronal  assembly  or  netlets  (see,  for  example,  Harth  et  al.,  1970;  Annios  et  ah, 
1970;  Usher  et  ah,  1993;  Edelman,  1987;  van  Vreeswijk  et  ah,  1976;  Wennekers 
et  ah,  1995  and  Wennekers  and  Pasemann,  1996).  A  netlet  consists  of  randomly 
interconnected  probabilistic  neurons,  and  netlet  behavior  is  described  in  terms 
of  the  activity  A(n)  €  [0, 1]  which  represents  the  fraction  of  neurons  firing  at 
any  discrete  integer  time  n  (Harth  et  ah,  1970;  Annios  et  ah,  1970;  Usher  et  ah, 
1993).  Discrete-time  dynamics  are  justified  by  refractoriness,  and  the  temporal 
and  spatial  fine  structure  in  neuronal  activity  within  a  netlet  are  considered  to  be 
of  secondary  importance  and  subsumed  by  netlet  dynamics. 

We  observe  that  plots  of  A(n+ 1)  vs.  A(n)  typically  obtained  in  netlet  analysis, 
(Harth  et  ah,  1970;  Annios  et  ah,  1970;  Usher  et  ah,  1993),  bear  remarkable 
similarity  to  the  return  map  of  the  logistic  (quadratic)  map  (Hilbom,  1994).  This 
suggests  that  the  basic  functional  unit  for  higher-level  cortical  processing  could 
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be  represented  and  modeled  by  a  nonlinear  iterative  map  on  the  unit  interval  and 
that  networks  of  parametrically  coupled  logistic  maps ,  resembling  those  studied 
also  in  (Farhat  et  al.,  1996;  Perez  et  al.,  1989  and  Bertille,  1990),  can  be  used  to 
model  and  investigate  cortical  dynamics  and  therefore  higher-level  brain  function. 
Similar  work  on  coupled  map  lattices  was  carried  out  by  Crutchfield  and  Kaneko 
(1987)  and  Kaneko  (1993).  The  state-space  orbits  or  trajectories  X (n)  produced 
by  the  logistic  map,  like  other  nonlinear  mapping  on  the  unit  interval,  are  known 
to  exhibit  periodic  (period-m)  orbits,  chaotic  orbits,  and  bifurcation  between  them 
depending  on  the  value  of  the  control  or  bifurcation  parameter  p  of  the  map.  The 
control  parameter  can  be  made,  for  example,  as  in  (Farhat  et  al.,  1996),  to  be  the 
input  or  driving  signal  of  the  map  resulting  in  a  parametrically  driven  logistic 
processing  element.  One  can  expect  netlets  to  behave  in  a  like  manner  where 
the  activity  A(n),  analogous  to  X(n)  of  the  logistic  map,  would  exhibit  similar 
complex  orbits.  Indeed,  such  behavior  has  been  observed  in  some  of  the  netlet 
models  studied  (Wennekers  and  Pasemann,  1996).  What  can  be  quite  significant, 
is  that  periodicity,  bifurcation,  and  chaos  can  emerge  on  the  driven  netlet  level 
despite  the  well  known  imprecision  of  neuronal  firing.  This  picture  goes  some 
way  towards  elucidating  the  neuronal  code  and  the  way  brain  development  and 
evolution  succeeded  in  getting  precise  (repeatable)  higher-level  cortical  functions 
from  relatively  imprecise  neural  wetware.  It  provides  a  plausible  answer  to  an 
important  question  in  neuroscience,  namely  how  coherence,  synchronicity,  phase¬ 
locking,  and  deterministic  chaos  can  operate  on  the  netlet  and  cognitive  levels 
despite  the  imprecise  nature  of  neuronal  firing. 

Work  described  in  this  paper  is  therefore  based  on  the  novel  and  intriguing 
preposition  that  the  basic  functional  unit  for  higher-level  processing  in  cortical 
networks  can  be  modeled  by  a  bifurcation  processing  element  (PE)  like  the  para¬ 
metrically  driven  logistic  map  (Farhat  et  al.,  1995)  or  any  other  one-dimensional 
driven  map  on  the  unit  interval.  The  important  attribute  of  such  a  PE,  which 
distinguishes  it  from  sigmoidal  processing  elements  (sigmoidal  model  neurons) 
used  extensively  in  present  day  neural  networks,  is  the  complex  way  it  encodes 
information  it  receives  in  the  form  of  modulation  of  its  control  parameter  caused 
by  inputs  from  other  PEs  in  a  network.  (The  behavior  of  a  bifurcation  PE  is  best 
described  by  a  bifurcation  diagram  hence  the  name.)  In  this  scheme,  a  cortical 
net  of  N  interacting  netlets,  each  of  which  made  of  a  large  number  (hundreds 
or  thousands)  of  probabilistic  neurons,  can  be  simply  modeled  and  simulated  by 
a  network  of  N  parametrically  coupled  bifurcation  processing  elements.  Earlier 
work  (Farhat  et  al.,  1995;  Farhat  et  al.,  1994),  indicates  that  bifurcation  PEs  can 
be  realized  in  analog  hardware  by  an  integrate  and  fire  model  neuron  equipped 
with  a  dendritic  tree  when  it  is  subjected  to  correlated  spike  trains  at  its  dendrites 
that  give  rise  to  a  periodic  activation  potential  at  its  hillock.  We  may  have  there¬ 
fore  available  to  us,  for  the  first  time,  the  means  for  modeling,  simulating,  and 
studying  cortical  networks  both  in  software  and  hardware.  What  all  this  implies, 
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and  can  be  quite  remarkable,  is  that  in  simulation  studies  a  net  of  N  bifurcation 
PEs  would  effectively  be  equivalent  to  (102  to  103)  x  N  cortical  neurons. 

3.  Parametrically  Coupled  Network  of  Logistic  PEs  (The  Logistic  Net) 

The  architecture  of  the  logistic  net,  shown  in  Figure  1,  consists  of  N  paramet¬ 
rically  coupled  logistic  PEs  (parametrically  driven  logistic  maps)  described  by 
Xi(n  +  1)  =  m(n)Xi(n)[  1  -  Xi(n)},  where  ^i(n)  €  [0,4],  and  Xt (n)  G  [0, 1], 
i  =  1 , 2, . . . ,  N,  together  with  N  auxilliary  “sensory”  logistic  maps  described 
by  Xf(n  +  1)  =  iJfXf(n)[  1  -  Xf(n)]  with  fif  G  [0,4]  and  X?  G  [0,1].  The 
sensory  network  is  used  to  produce  a  wide  range  of  spatio-temporal  input  pat¬ 
terns  to  drive  the  network.  The  network  employs  nonlinear  activity  dependent 
coupling  functions  between  PEs  as  an  abstraction  of  the  possible  occurrence  of 
rapid  activity  dependent  modulation  of  the  coupling  between  netlets. 

To  loosely  incorporate  this  activity  dependent  coupling  in  the  network  of 
Figure  1  we  use  binned  or  quantized  nonlinear  coupling  functions  Bf  (Xf )  from 
the  sensory  elements  to  the  PEs  and  Bij(Xj)  between  the  processing  logistic 
elements  (the  yth  and  7th  elements)  respectively.  The  step-like  nature  of  these 
binned  coupling  functions  illustrated  in  Figure  2  allows  sudden  changes  in  the 
coupling  strengths  between  netlets  as  their  activity,  represented  here  by  the  state 
variable  Xi(n )  of  the  logistic  PE,  changes.  The  number  of  levels  (steps)  in 
the  binned  coupling  functions  is  arbitrarily  chosen  to  be  4.  The  use  of  binned 
coupling  is  found  to  lead  to  the  emergence  of  clustering  and  to  the  avoidance 
of  “fragmentation”  in  network  activity.  As  seen  in  Figure  2,  the  binned  coupling 
function  By  (Xj )  for  example  is  formed  by  the  intersection  of  the  horizontal  lines 

A,  T,  C,  G  and  the  nonlinear  function  gtj(Xj )  =  4xfij,  where  Cy  is  a  positive 
real  constant  that  determines  the  shape  of  gij{Xj )  and  therefore  the  location  of 
the  steps  or  breakpoints  a,  b,  c.  Similar  rules  apply  to  Bf(Xs). 

The  binned  coupling  functions  B?(XS)  and  By(Xj)  determine  the  value  of 
the  control  parameter  /y(n)  of  the  Ah  PE  in  the  network  according  to, 

IM(n)  =  eOf  (n)  +  ^  O  'y(n),  (1) 

3= 1 

where  0?(n)  =  B?(X?{n))  and  Oy(n)  =  By(X,(n))  with  Xf(n)  and  Xj(n) 
being  the  state  variables  of  the  driven  sensory  and  processing  logistic  maps 
respectively  and  e  is  a  parameter  controlling  the  relative  effects  of  the  extrinsic 
(sensory)  and  intrinsic  (feedback)  activity  on  The  simulations  presented 

here  are  for  e  =  0.5. 

We  have  carried  out  (Farhat  et  al.,  1996)  a  numerical  simulation  of  the  logistic 
net  of  Figure  1  for  N  =  100  under  a  variety  of  stimulus  generating  vectors  JLS. 
Homogeneous  binned  coupling,  i.e.,  Cf  =  Cy  =  0.5,  i,j  =  1,2,  ...,1V,  was 
used  because  the  emphasis  was  on  studying  how  the  network  behaves  under 
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Of  (n)  =  Bf  (xf(n)) ,  Oij(n)  =  By(Xj(n)) 

Figure  1.  Parametrically  coupled  net  of  logistic  processing  elements  (PEs)  with  binned 
(quantized)  coupling  functions. 

different  types  of  sensory  input  patterns  and  not,  in  this  paper,  on  adaptation  and 
learning  where  Cij  would  be  incrementally  altered  for  example  by  a  suitable 
adaptation  algorithm  to  effect  learning.  In  the  simulations  described  here,  all 
logistic  PEs  in  the  net  were  initiated  from  the  same  arbitrarily  chosen  initial  state 
X/(o)  =  Xi( o)  =  0.5.  In  Figure  1,  X(n)  is  the  state  vector  of  the  network  and 
ps  is  the  input- generating  vector .  Note  that  depending  on  the  nature  of  ~jJs  (or 
its  components  pf)  the  network  can  be  driven  by  coherent  (periodic),  partially 
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Figure  2.  Nonlinear  quantized  (binned)  coupling  functions,  (a)  Bi(Xf)  between  the  sensory 
logistic  elements  and  the  network’s  PEs,  and  (b)  Bij(Xj)  between  PEs. 


coherent  (partly  periodic  and  partly  chaotic),  or  chaotic  input  sequences  (orbits) 

B!(X?(n))  =  B?(n)  =  Of  (n). 

4.  Simulation  Results 

In  this  section  we  summarize  the  results  of  numerical  experiments  carried  out 
to  date  with  the  logistic  net  of  Figure  1.  To  characterize  the  behavior  of  the 
parametrically  coupled  logistic  net  we  have  introduced  the  concept  of  limit-set- 
diagram  (LSD).  The  LSD  is  a  static  representation  of  the  state  vector  X(n )  of 
the  network,  i.e.,  of  the  orbits  Xi(n),  i  =  1,2 , ,N,  of  the  PEs.  It  is  formed 
by  entering  the  values  Xi(n )  €  [0, 1]  visited  by  the  post-transient  orbit  above 
each  integer  point  i  =  1 , 2, . . . ,  N  representing  PE  number  taken  as  the  abscissa 
of  the  diagram  (see  bottom  row  of  Figure  3).  The  top  row  of  Figure  3,  shows 
three  different  input  generating  vectors  JL s.  Those  in  (a)  and  (b)  produce  coherent 
stimulus  vectors  Xf{n )  and  associated  Bf  (n)  because  none  of  the  components 
/if  of  the  input  generating  vector  are  chaos  inducing,  i.e.,  /if  <  /j.c  =  3.56.  Here 
fic  =  3.56  is  the  critical  value  of  the  control  parameter  of  a  logistic  map  above 
which  chaos  is  permitted.  In  both  (a)  and  (b)  the  inputs  to  the  network,  Xf{n) 
or  Bf(n),  are  therefore  ordered  and  void  of  chaos.  In  (c)  on  the  other  hand  Jis 
has  chaos  inducing  components  where  /if  >  3.56.  As  a  result,  both  Xf  (n)  and 
Bf(n)  have  few  chaotic  components  (93  <  i  <  100)  and  the  input  to  the  network 
in  this  case  is  partially  coherent. 

The  LSDs  associated  with  each  of  these  inputs  are  given  in  the  bottom  row 
of  Figure  3.  All  these  are  seen  to  exhibit  clustering  wherein  the  PEs  cluster 
into  groups  of  identical  orbits.  The  LSDs  in  (a)  and  (b)  represent  instances  of 
the  network  converging  to  a  periodic  attractor  of  period-m  and  low  dimension 
Nc,  where  Nc  <C  N  is  the  number  of  clusters  (Nc  =  6  in  (a)  and  Nc  =  7  in 
(b)).  Analysis  of  individual  orbits  of  the  PEs  revealed  that  all  those  belonging 
to  the  same  cluster  have  synchronized  orbits  i.e.,  identical  orbits.  Each  cluster 
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acts  then  as  a  single  unit,  i.e.,  like  a  “super-neuron”.  All  six  clusters  are  found 
to  be  phase-locked,  in  that  they  have  distinct  periodic  orbits  that  are  in  fixed 
relative  phase  and  of  the  same  period  -  m.  We  found  m  =  4  in  (a)  and  m  —  8 
in  (b).  Convergence  to  a  periodic  attractor  is  sudden  following  an  erratic  orbit 
interval  of,  for  example,  1700  iterations  for  the  case  in  (b).  During  the  transient 
period,  the  network  searches  its  state  space  for  a  “loss-region”  in  which  it  can 
collapse  into  a  low-dimensional  periodic  attractor  compatible  with  the  constraints 
on  the  network  namely  the  applied  input  and  the  coupling  functions  matrix. 
Similar  analysis  of  PE  orbits  for  the  case  in  (c)  shows  the  orbits  do  not  converge 
to  a  periodic  attractor  even  after  observation  over  few  thousand  iterations,  but 
appear  instead  to  converge  to  a  strange  attractor  by  virtue  of  the  distinct  structure 
of  the  LSD  in  (c).  In  the  LSD  in  (c),  there  appear  to  be  a  total  Nc  =  10 
clusters  out  of  which  8  consist  of  two  or  more  PEs.  Recall  the  input  generating 
vector  Jis  in  (c)  had  chaos  inducing  components  for  which  fxf  >  fic  =  3.56. 
All  this  seems  to  suggest  that  when  the  input  generating  vectors  considered  here 
produce  incoherent  or  partially  coherent  input  patterns  our  net  never  converges 
to  a  periodic  attractor  but  seems  to  converge  instead  to  a  strange  attractor.  One  is 
tempted  to  hypothesize  that  convergence  to  a  periodic  attractor  in  dynamical  nets 
is  synonymous  with  recognition  and  classification  of  the  input;  while  convergence 
to  a  strange  attractor  indicates  lack  of  recognition  because  the  input  pattern  pre¬ 
sented  to  the  net  is  not  completely  coherent  (it  contains  incoherent  (meaningless) 
components  that  do  not  correlate  with  other  components  of  the  feature  vector 
(taken  to  be  JLS),  and  this  prevents  the  network  from  linking  the  various  parts  of 
the  feature  vector  into  a  coherent  unit  manifested  by  the  convergence  to  periodic 
attractor.  Close  examination  of  the  orbits  Xi(n)  belonging  to  different  clusters 
in  this  latter  case  shows  they  are  not  phase-locked  because  they  do  not  exhibit 
any  periodicity  even  over  extended  observation  windows  of  5000  iterations  and 
more.  This  encourages  us  to  suggest  that  phase-locking  of  clusters  maybe  used 
as  mechanism  of  feature  binding  in  dynamical  networks  such  as  the  logistic  net 
studied  here. 


5.  Discussion 

Dynamical  nets,  like  the  logistic  net  discussed  here,  show  behavior  consistent 
with  the  general  idea  in  several  cortical  oscillations  theories  (von  der  Marls- 
burg,  1981,  1986;  Abeles,  1982),  and  with  the  discovery  of  long-range  temporal- 
correlation  in  cortical  activity  in  cat  and  monkey  (Eckhom  et  al.,  1988;  Gray, 
1989;  Singer,  1993;  Bosch  et  al.,  1995;  Engel  et  al.,  1991(a),  1991(b);  Murthy 
et  al.,  1992).  At  this  stage  of  our  work  on  dynamical  nets  we  can  make  the 
following  remarks  that  could  be  relevant  to  cortical  dynamics  and  higher-level 
processing: 
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•  Externally  driven  networks  of  parametrically  coupled  logistic  processing  ele¬ 
ments  constitute  a  computationally  efficient  setting  for  the  simulation  and 
study  of  dynamical  neural  networks  and  dynamical  computing. 

•  Such  networks  compute  with  diverse  attractors  (point,  periodic  and  chaotic 
(strange))  and  are  capable  of  bifurcating  between  attractors  in  the  course  of 
computation. 

•  In  dynamical  networks,  bifurcation  which  is  a  sudden  qualitative  change  in 
network  behavior  as  a  result  of  change  in  a  control  parameter  or  input,  is  a 
source  of  diversification  and  innovation  that  can  lead  to  new  solutions. 

•  Dynamical  networks  are  capable  of  conducting  transient  chaotic  search  of 
their  state  space  for  attractors  compatible  with  their  input. 

•  The  nonlinear  (quantized  or  binned)  coupling  between  processing  elements 
used  here  allows  rapid  changes  in  the  coupling  between  PEs  to  take  place.  It 
is  introduced  to  loosely  reflect  the  possibility  that  coupling  between  netlets 
is  activity  dependent  and  can  change  abruptly.  We  also  find  that  binned 
coupling  prevents  fragmentation  where  the  number  of  clusters  (groups  of 
synchronized  PEs)  becomes  very  large. 

•  The  logistic  network  converges  to  a  periodic  attractor  or  to  chaotic  (strange) 
attractor  depending  on  its  input.  When  the  input  is  coherent,  the  network  col¬ 
lapses  suddenly,  following  a  transient  search  interval,  to  a  low  dimensional 
manifold,  a  periodic  attractor  associated  with  the  particular  input.  When  the 
input  is  incoherent  or  partially  coherent  it  does  not  converge  to  a  periodic 
attractor  but  seems  to  converge  instead  to  a  strange  attractor. 

•  The  dimension  of  the  periodic  attractor  equals  the  number  of  clusters  Nc, 
and  Nc  <C  N.  It  is  as  if  the  network  partitions  itself  into  a  small  number  of 
subpopulations  (clusters)  with  all  “PEs”  within  a  cluster  having  exactly  the 
same  orbit,  i.e.,  have  synchronized  orbits.  The  clusters  are  phase-locked  and 
each  cluster  can  be  viewed  as  a  “super  neuron”. 

•  A  small  number  of  PEs  (100  in  the  network  simulated)  can  participate  in  a 
very  large  number  of  distinct  convergent  states  depending  on  the  constraints 
on  the  network  (input  and  coupling  between  neurons).  A  relatively  small  net 
may  be  used  therefore  in  the  classification  of  a  very  large  number  of  input 
patterns. 

•  When  the  input  is  incoherent  we  observe  clustering  with  synchronicity  within 
a  cluster,  but  the  clusters  are  not  phase-locked. 

•  If  we  regard  the  input  JIS  to  the  dynamical  (logistic)  net  considered  here  as  a 
feature  vector  generator,  then  when  the  feature  vector  is  coherent,  because  ~ps 
has  no  chaos  inducing  components,  i.e.  (the  input  to  the  network  will  consist 
of  correlated  or  meaningful  components  (features)),  the  network  would  con¬ 
verge  to  a  periodic  attractor  performing  thereby  a  feature  binding  operation. 
When  the  input  is  incoherent  or  partially  coherent,  meaning  the  components 
of  the  feature  vector  are  not  compatible  with  one  another,  the  network  con- 
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verges  to  a  chaotic  or  strange  attractor.  The  logistic  net  studied  is  capable 
therefore  of  performing  a  feature-binding  or  feature-linking  operation. 

•  Coherence,  phase-locking,  and  clustering  of  orbits  provide  the  basis  for  a 
formalism  for  neuroholography  (dynamic  kinoform  concept)  because  the 
state  variable  7Q(n)  G  [0, 1],  i  =  1 , 2, . . . ,  N,  of  the  network  can  be  regarded 
as  a  normalized  modulu  27t  phase  variable.  Note  kinoform  is  the  term  used 
in  optics  to  describe  a  phase-only  hologram. 

•  The  form  of  the  nonlinear  coupling  function  between  neurons,  specifically 
the  coupling  factor  Cy,  permits  study  and  exploration  of  learning  algorithms 
for  dynamical  networks  possibly  based  on  information  theoretic  measures  of 
the  orbits  Xj(n)  like  mutual  information  between  orbits. 

As  stated  earlier,  the  logistic  net  described  and  studied  here  is  similar  to  Kaneko’s 
coupled  map  lattices  (Kaneko,  1993;  Crutchfield  and  Kaneko,  1987).  There  are 
however  three  significant  differences.  One  is  the  nonlinear  and  global  nature 
of  the  coupling  used  here,  two  is  the  use  of  parametric  coupling  rather  than 
diffusive  coupling  and  three  is  the  provision  of  auxilliary  sensory  elements  for 
generating  complex  spatio-temporal  input  signals.  Such  provision  of  extrinsic 
input  capability  extends  the  scope  of  modeling  applications  of  parametrically 
coupled  nets  of  logistic  or,  other  bifurcation  processing  elements,  beyond  those 
listed  above.  For  example,  one  can  study  the  behavior  of  such  nets  when  the 
parameter  e  in  Equation  (1)  is  made  to  slowly  decay  exponentially  in  time  in  order 
to  pass  the  control  over  network  dynamics  from  initially  extrinsically  dominated 
control  to  eventually  completely  intrinsic  control.  The  effect  of  the  extrinsic  input 
in  this  scenario  is  to  conduct  the  network  to  specific  region  of  its  state-space  when 
all  the  while  it  is  being  gradually  released  to  entirely  intrinsic  control.  Such 
behavior  maybe  plausible  in  biological  networks  (Freeman,  1995)  and  may  also 
be  relevant  to  the  development  of  learning  algorithms  for  dynamical  networks. 
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